Evaluating Socio-Technical Systems with Heuristics - A Feasible Approach?

In the digital world, human centered technologies are becoming more and more complex socio-technical systems (STS) than in previous years thus challenging existing traditional design and evaluation methods. There are two new tendencies, a) STS-methods for design and development are complemented by methods for evaluation of existing STS, and b) the question arises on how to evaluate STS and their quality sufficiently. In this paper we illustrate a method to develop STS heuristics combining four differed areas of research: CSCW, job design, usability heuristic and socio-technical design principles. Initial results will be presented and discussed. 1 Introduction Over the past years, human centered technology and socio-technical projects have emerged that challenge existing evaluation methods. Project examples are wearables in health care, remote service systems for older adults, applications of cyber physical systems and others. A major part of those design projects is their evaluation. This part of the design process is especially crucial since it becomes more and more evident that it is not possible to plan all aspects of complex socio-technical systems (STS) in advance. Those systems rather require a mutual adaptation process between technology and the context they are used in [1–3] which requires suitable methods of evaluation. Usability testing is a well-established procedure for evaluating interactive systems in this context and a variety of evaluation methods has been developed in the past. Within these methods, heuristics have become a recognized approach to identify usability problems. Heuristics can serve as a basis of cognitive walkthroughs where experts inspect the features of an interactive system step-by-step. Typical heuristics like the ones by Nielsen (c.f. Fig. 1) or by the International Standard Organization (c.f. ISO 9241-110, 2006) cover aspects such as the suitability of a system for a task, its controllability, individualization, self-descriptiveness, conformity with user expectations , error tolerance, and suitability for learning. In this contribution, we argue that it is useful to employ heuristics also in order to evaluate STS. This however requires existing usability heuristics to be revised or adjusted in order to cover the much broader context of STS. While usability issues are focusing mainly on human-computer-interaction (HCI) and individuals, STS are more complex systems in the sense that they involve HCI and team work which is mediated by technology. Consequently, our research questions are: To what extent and in which